Consolidate shared scraping base (update) - #21

Taken verbatim from MedARC-AI#10 so the two merge cleanly in either order. A listing
entry is not always a document: PMC holds some open-access records as metadata
only, with no full text deposited. Returning None skips the entry without
consuming the --documents budget, and the delay gate moves from scraped to
attempted so skipped items still rate-limit.

Drop this commit if MedARC-AI#10 lands first.

Scrapes the intersection of PubMed's Guideline publication type with the PMC
Open Access subset, restricted to English: about 3,000 guidelines. That is the
only slice where the full text is both retrievable and openly licensed; the
other 40k Guideline records expose an abstract only, under publisher copyright.

Two scoping choices, both measured rather than assumed:

`Guideline[pt]` rather than `Practice Guideline[pt]`. The former is a strict
superset and the 352 records it adds are clinical, not administrative (the 2025
Korean CPR guidelines and similar), so the narrower tag would drop 11% of the
corpus for nothing.

`English[la]`, which drops 185 records. Every other source in this package is
already English-only as a side effect of its entry URL: CPS is a bilingual site
scraped through its /en/ routes, WHO publishes in six languages and is scraped
through its English listing. PubMed's API returns every language, so the filter
has to be explicit to match. The excluded records are largely French CMAJ
translations of guidelines already in the corpus.

Discovery and extraction use NCBI's E-utilities rather than the rendered pages.
esearch paginates by numeric offset, so the page number maps straight onto
retstart and an empty page past the end terminates the run. esummary resolves a
whole batch of PMIDs to PMCIDs and citation metadata in one request. efetch
returns JATS XML carrying body, section structure and license.

JATS is close enough to HTML that renaming tags and reusing html_to_markdown is
cheaper and less error-prone than a second serializer: table-wrap already
contains genuine XHTML tables, and inline markup maps one to one.

Licensing is recorded per document rather than claimed for the source, because
the Open Access Subset is not uniformly Creative Commons licensed. Censused
over all 2,999 English records present on 2026-07-29:

  CC BY          44.5%    publisher terms, no CC license   11.7%
  CC BY-NC       22.1%    of which: Elsevier COVID grant, no
  CC BY-NC-ND    18.4%              <license> element at all (112),
  CC BY-NC-SA     2.1%              PMC OA "unrestricted re-use"
  CC0             1.3%

By what that permits: 45.8% unrestricted for derivative works, 24.1%
non-commercial only, 18.4% asserting NoDerivatives, 11.7% needing a
case-by-case reading. Presence in the subset is not itself a grant to
redistribute: 112 records carry only a copyright line such as "(c)
Springer-Verlag Tokyo 2007", and Elsevier's pandemic-era deposits grant free
access while still reserving all rights.

The license name is parsed from the Creative Commons URL rather than the
license-type attribute, which the corpus spells 18 different ways. The
copyright statement is captured separately because it is a sibling of
<license>, not a child, and it holds the reservation of rights.

Figures and supplementary files are recorded in metadata rather than linked.
Unlike the HTML sources in MedARC-AI#9 and MedARC-AI#10, which absolutize a real <img src>, JATS
carries only a bare filename; the served URL inserts a CDN shard and content
hash that appear nowhere in the API response, so a constructed link 404s.
Supplementary blocks are pointers too: across 40 sampled guidelines every one
referenced an external .docx or .tif rather than inline content, totalling
0.18% of body text. Recording name, label and caption keeps the evidence
tables findable without re-scraping.

E-utilities calls retry with backoff on 429 and 5xx. One document makes up to
two calls back to back and the rate limit is per source address, so NCBI does
answer with 429 in practice; without a retry that propagates past the skip
handler and kills the whole run.

external_id prefers the PMID from the record itself, so an article reached
from a PMC URL gets the same identifier as one reached from the listing.

Records PMC holds without a deposited body, 0.6% of the corpus, are logged
and skipped rather than aborting the run.

Taken verbatim from MedARC-AI#10 so the two merge cleanly in either order. A listing
entry is not always a document: PMC holds some open-access records as metadata
only, with no full text deposited. Returning None skips the entry without
consuming the --documents budget, and the delay gate moves from scraped to
attempted so skipped items still rate-limit.

Drop this commit if MedARC-AI#10 lands first.

(cherry picked from commit b55d196)

Taken as written from the CPS and RCH scraper PRs by mkrastev (MedARC-AI#9, MedARC-AI#10), where this change appears identically inside larger commits and could not be cherry-picked on its own.

Leaves only the base.py and html.py changes the source branches share.
Each source PR adds its own module, tests and licence notes when it merges.

Registering a source was an enum member, a match arm, an _expand_source entry
and an __init__ re-export block, in two files. It is now one entry in SCRAPERS.

Also fixes --source all, which returned inside the loop and so only ever ran
the first source.

A source whose client is not httpx (Playwright, for one) can drive the same
loop. The loop passes the client through untouched, so this is types only.

urljoin turned '#recommendations' into the page URL, so a link to a heading in
the same document pointed at the document instead.